Method, system and computer program product for sampling computer system performance data

ABSTRACT

A system, method and computer program product for sampling computer system performance data are provided. The system includes a sample buffer to store instrumentation data while capturing trace data in a trace array, where the instrumentation data enables measurement of computer system performance. The system further includes a sample interrupt generator to assert a sample interrupt indicating that the instrumentation data is available to read. The sample interrupt is asserted in response to storing the instrumentation data in the sample buffer.

BACKGROUND OF THE INVENTION

This invention generally relates to computer system performancemonitoring. More specifically, the invention relates to samplingcomputer system performance data without impacting reliability,availability and serviceability of the computer system.

Computer system performance measurement enables detection of issues thatcan result in reduced throughput of the computer system. One approach tomeasuring performance is to repeatedly execute workload instructionstreams, which are often segments of customer workload code targeted tostress particular hardware and/or software functions, and collect datarelevant to the system's performance. Initially, hardware capturesselected signals and stores them in hardware arrays for furtheranalysis. Each group of the selected signals is called a “sample”. Whenenough samples have been captured to fill the arrays, a hardwareinterrupt invokes firmware to move the data from the arrays to storage.A set of controls provides flexibility for a user (e.g., a measurementteam member) in selecting which signals are captured and when theselected data is captured. The captured data are later used forcalculating performance analysis metrics such as cycles per instruction(CPI), cache misses/hits, pipeline stalls, and the like. Basicmechanisms for data capturing and performance measurement, also referredto as “instrumentation”, are described in U.S. Pat. Nos. 4,590,550, and4,821,178, each of which is hereby incorporated herein by reference inits entirety.

Historically, to reduce hardware footprint, instrumentation has takenadvantage of hardware arrays already existing in a design. These arrayswere originally intended for hardware tracing to capture machine statesover a period of time for debug data. When a failure occurs, the data inthe arrays, once extracted, serve as a record of events leading up tothe failure. Along with providing debug data in a lab environment,hardware tracing can greatly enhance computer system serviceability. Inthe event of a failure in the field (e.g., customer location), thiscapability facilitates problem isolation and resolution. Further, once aproblem is understood, design changes can be implemented to improvefuture reliability. Hardware controls, such as multiplexers, areprovided to allow the user to select which signals are routed to thehardware arrays. Several hardware tracing modes may be defined to assistin debugging particular scenarios, and the multiplexers provideswitching between the modes. A further mode is defined to facilitateinstrumentation. Different sets of signals are routed, via themultiplexers, to the hardware arrays for hardware tracing andinstrumentation. The instrumentation signals can be used for evaluatingsystem performance.

Additional controls are provided to define events, upon which to capturedata. Typical settings for collecting debug data using hardware tracinginclude starting on an instruction address. For instrumentation, it isdesirable to start collecting data on a time increment and to capture aset of data on regular time intervals. During a typical instrumentationrun, the multiplexers are set to route instrumentation signals to thehardware arrays, and the event controls are set to collect data on atime interval. Each time the interval expires, the instrumentationsignals are captured and saved in the next available row within thehardware arrays. When all rows of the arrays have been filled, aninterrupt invokes firmware to copy the data to a buffer in storage. Uponexiting the firmware routine, the run continues.

A major drawback to this existing approach is that runninginstrumentation entails switching the hardware controls intoinstrumentation mode, thereby disabling hardware tracing. Withouthardware tracing enabled, failure analysis is extremely difficult, andreliability/availability/serviceability (RAS) is compromised. Thus,instrumentation is not typically run in the field at a customer site dueto the resulting reduction in RAS.

It would be beneficial to allow instrumentation data to be capturedwhile simultaneously running hardware tracing. To keep complexity andhardware costs to a minimum, it would be desirable to supportinstrumentation and hardware tracing without duplicating the entirecollection of hardware currently used for hardware tracing.Additionally, it would be advantageous to allow sampling ofinstrumentation data in the field without reducing RAS. Accordingly,there is a need in the art for sampling computer system performance datawithout impacting RAS of the computer system.

BRIEF SUMMARY OF THE INVENTION

An exemplary embodiment includes a system for sampling computer systemperformance data. The system includes a sample buffer to storeinstrumentation data while capturing trace data in a trace array, wherethe instrumentation data enables measurement of computer systemperformance. The system further includes a sample interrupt generator toassert a sample interrupt indicating that the instrumentation data isavailable to read. The sample interrupt is asserted in response tostoring the instrumentation data in the sample buffer.

Another exemplary embodiment includes a method for sampling computersystem performance data. The method includes sampling instrumentationdata in a sample buffer while capturing trace data in a trace array,where the instrumentation data enables measurement of computer systemperformance. The method also includes asserting a sample interrupt inresponse to sampling the instrumentation data. The sample interruptindicates that the instrumentation data is available to read.

A further exemplary embodiment includes a computer program product forsampling computer system performance data. The computer program productincludes a computer-readable storage medium storing instructions forexecuting sampling computer system performance data. The samplingcomputer system performance data includes a method of receiving anindicator of a sample interrupt, where the sample interrupt indicatesthat instrumentation data is available to read from a sample buffer formeasurement of computer system performance. The method also includesselecting a data source to read as the sample buffer in response toreceiving the indicator of the sample interrupt, where the sample bufferstores instrumentation data while capturing trace data in a trace array.The method further includes storing data from the selected data sourceto memory.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings wherein like elements are numbered alikein the several FIGURES:

FIG. 1 depicts a block diagram of system upon which sampling performancedata without impacting RAS of the system can be implemented in anexemplary embodiment;

FIG. 2 depicts a block diagram of trace array and logic in accordancewith an exemplary embodiment;

FIG. 3 depicts a block diagram of a sample pulse generator and a sampleinterrupt generator in accordance with an exemplary embodiment; and

FIG. 4 depicts an exemplary process for sampling computer systemperformance data without impacting RAS of the computer system inaccordance with an exemplary embodiment.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

An exemplary embodiment of the present invention provides computersystem performance data sampling without impactingreliability/availability/serviceability (RAS) of the computer system.Through a combination of both hardware and firmware, instrumentation canbe run without disabling hardware tracing in the computer system.Instrumentation allows samples of selected signals that are critical toperformance measurement to be captured. A separate set of signals isavailable to collect debug data for hardware tracing. In an exemplaryembodiment, both instrumentation and hardware tracing are concurrentlysupported. Using a set of latches to hold one sample of instrumentationdata and a separate interrupt mechanism results in a minimal additionalhardware cost for the improved RAS performance. Additionally, existinginstrumentation support can remain in the design to provide both alegacy control mode and a new control mode.

Turning now to FIG. 1, a system 100 is depicted upon which samplingperformance data without impacting RAS of the system is implemented inan exemplary embodiment. The system 100 includes multiple trace arrayand logic blocks 102, which may be physically distributed in hardware.For example, in one embodiment the system 100 is a microprocessor chip,where the trace array and logic blocks 102 are spread around on the chipsurface or located in some designated part of the chip, such as themicroprocessor core. In a single microprocessor, there may be ten ormore instances of trace array and logic blocks 102, supporting a widthof up to sixty-four or more bits for each. In an exemplary embodiment,the trace array and logic blocks 102 are spread out to place them nearbypoints of origination of the signals to be captured.

The system 100 also includes a trace controller 104 to control readingand writing data to/from the trace array and logic blocks 102. Data fromthe trace array and logic blocks 102 are returned to a trace datacollector 106. Control signals are sent to the trace array and logicblocks 102 via paths 108 and 110. In an exemplary embodiment, path 108sends signals via a fanout tree using repeaters/latches as required forthe physical signal distribution to all interfaced trace array and logicblocks 102, while paths 110 provide individual control signals to theinterfaced trace array and logic blocks 102. The paths 108 and 110 maybe single bit or multi-bit busses, controlling sampling ofinstrumentation data input 112 and capturing of trace data input 114.Each of the trace array and logic blocks 102 can read bothinstrumentation data input 112 and trace data input 114. The trace arrayand logic blocks 102 output data on data return paths 116 to the tracedata collect block 106. It will be understood that paths 108 and 110, aswell as data return paths 116, can include elements known in the artthat are not depicted, such as latches to accommodate for relativetiming differences.

In an exemplary embodiment, the trace controller 104 interfaces with aninterrupt register 118 and firmware 120 via paths 122 and 124respectively. The interrupt register 118 provides information such as aninterrupt source when an interrupt is asserted. The firmware 120 islow-level control code that provides instruction sequences forcontrolling instrumentation and hardware tracing operations, e.g.,millicode. The firmware 120 can access the interrupt register 118 viapath 126. The paths 122, 124, and 126 can be combined in anycombination, including a single common bus. The trace controller 104 mayalso include sample pulse generator (SPG) 128 that generates samplingsignals for controlling acquisition of the instrumentation data input112 in the trace array and logic blocks 102. The trace controller 104can also include a sample interrupt generator (SIG) 130 and a log traceinterrupt generator (LTIG) 132 among other logic blocks (not depicted).Alternatively, the SPG 128, SIG 130, and/or LTIG 132 can be locatedexternal to the trace controller 104. The SIG 130 can assert a sampleinterrupt (SI), which is captured as SI indicator 134 in the interruptregister 118. Similarly, the LTIG 132 can assert a log trace interrupt(LTI), which is captured as LTI indicator 136 in the interrupt register118.

Turning now to FIG. 2, a block diagram of one of the trace array andlogic blocks 102 of FIG. 1 is depicted in accordance with an exemplaryembodiment. Instrumentation data input 112 and trace data input 114 areboth input to a data input multiplexer (mux) 202. The data input mux 202controls the source of input for capturing data in trace array 204.Using data select 206, the data input mux 202 selects between multipleinput buses that can be routed to the trace array and logic blocks 102,as a limited set of data may be required to debug a specific problem. Inan alternate embodiment, the data input mux 202 is omitted. The dataselect 206 is driven by the firmware 120 from a configuration register(not depicted) associated with each trace array and logic block 102.Alternatively, the data select 206 can be provided via path 108 or 110of FIG. 1, as design dictates. The trace array 204 includes multiplerows of storage, with each row wide enough to capture a sample ofmultiple signals. The capturing of data in the trace array 204 iscontrolled by the trace controller 104 of FIG. 1, which provides anarray row address 208 and write enable 210 via paths 108 or 110. Thearray row address 208 can be incremented every cycle on start event orperiodically on a time basis. The trace controller 104 determines howthe array row address 206 is incremented. When debug tracing is active,a continuously running trace is performed through writing selected tracedata input 114 to the trace array 204, row-by-row. Instrumentationsampling periodically, or on an event basis, captures one array row. Inan alternate exemplary embodiment, the data input mux 202 only selectsfrom trace data input 114, not instrumentation data input 112.

The instrumentation data input 112 is also routed to sample logic 212.The sample logic 212 includes a sample mux 214, which receivesinstrumentation data 112 as input as well as a feedback signal 216 fromsample buffer 218. Sample pulse 220 selects between the two inputs tothe sample mux 214 for sampling. In contrast to the trace array 204, thesample buffer 218 stores one sample at a time. This is enough for mostperformance analysis requirements, as instrumentation data input 112samples are taken slowly enough to allow reading the last sample beforea new one is taken. The sample buffer 218 may have an equivalent widthor a reduced width of a full array row in the trace array 204. A reducedwidth may be used since the number of signals that are interesting forperformance analysis can be considerably less than the number of signalsused for debugging using trace data.

To read the samples of the instrumentation data input 112, data returnpaths 116 of FIG. 1 can be used from the trace array and logic blocks102 to the trace data collector 106 of FIG. 1. Due to area and wiringconstraints, the data return paths 116 may not have the full array widthof the trace array 204. Instead, a byte mux 222 can used to reduce thepath width to one byte. The trace controller 104 of FIG. 1 provides abyte select 224 for controlling the byte mux 222, which can be sent viapaths 108 or 110. A combination of the byte mux 222 and a read mode mux226 allows instrumentation data samples from the sample buffer 218 to beplaced in the data return path 116 previously reserved for reading thetrace array 204. This provides an advantage in that read logic in thetrace controller 104 can be left unchanged, allowing bytes to be readfrom the sample buffer 218 in the same manner as reading from the tracearray 204. The row address is ignored, and instead of reading, forinstance 256 rows, just one sample is read, where a sample may be aswide as a single row.

The read mode mux 226 selects either instrumentation data from thesample buffer 218 or data from the trace array 204 as input to the bytemux 222. The read mode mux 226 is controlled from a latch 228, which mayin turn be set by decoding unused byte select 224 values asinstrumentation mode 230. For example, if byte select 224 contains eightpossible combinations, an unused combination can be used asinstrumentation mode 230. Thus the firmware 120 of FIG. 1 can set thebyte select 224 to a specified value. Alternatively, a separate singlebit control wire may be implemented as instrumentation mode 230 viapaths 108 or 110 of FIG. 1.

In summary, the sample buffer 218 can hold one sample of theinstrumentation data input 112 and operate in parallel with the tracearray 204. Regardless of the data input mux 202 setting, the samplebuffer 218 can always collect the instrumentation data input 112. Usingthe sample buffer 218, the instrumentation data input 112 can becaptured independent of the trace data input 114, allowing both hardwaretracing and instrumentation sampling to be run together for simultaneousperformance data collection and debug data collection. Additionally, thedata input mux 202 can be set to route either the instrumentation datainput 112 or the trace data input 114 to the trace array 204 as well,which can be useful for debugging and validating the hardware andfirmware.

To allow instrumentation and hardware tracing to run simultaneously, aseparate interrupt mechanism, SIG 130 of FIG. 1, is used forinstrumentation. Since the sample buffer 218 is one entry deep, thesample interrupt invokes the firmware 120 of FIG. 1 on every datacollection event. The firmware 120 determines the source of theinterrupt using the interrupt register 118 of FIG. 1 and respondsaccordingly. If the source is hardware tracing, LTI 136 is set, and thefirmware 120 selects and reads the trace array 204. Likewise, if theinterrupt source is instrumentation, SI 134 is set, and the firmware 120may store the instrumentation data to memory 144. The trace controller104 turns off array write controls and sends appropriate array rowaddress 208 and byte select 224 values through paths 108 and 110. Thisis triggered by the firmware 120 using path 124. The data that is readfrom the trace array 204 or sample buffer 218 is sent through path 116together with a “data valid” indication. This is all that is needed fortrace data collector 106 to start collecting the bytes of data.Interface 138 from the trace data collector 106 allows the data to beread to registers 140, which can be accessed by the firmware 120 viainterface 142. The firmware 120 stores the debug/sample data to memory144 via interface 146.

Turning now to FIG. 3, a block diagram of the SPG 128 and SIG 130 ofFIG. 1 are depicted in accordance with an exemplary embodiment. In anexemplary embodiment, the sample pulse 220 of FIG. 2 is generated byAND-gate 302 through combining output of a pulse generator 304 with aninverted value of latch 306. The pulse generator 304 may include aconfigurable timer to periodically generate a pulse for samplinginstrumentation data. An inverter 308 performs the signal inversion fromlatch 306. The latch 306 captures a state where a sample ofinstrumentation data input 112 was stored to the sample buffer 218 inFIG. 2, but not yet read by the firmware 120 of FIG. 1. The latch 306 isset from feedback from the sample pulse 220 and the value is held usingOR-gate 310 as long as no clear sample interrupt 312 is received viaAND-gate 314. In an exemplary embodiment, the clear sample interrupt 312is commanded through firmware interface 124 once a sample captured inthe sample buffer 218 of FIG. 2 has been read. The latch 306 alsosources the SIG 130 to assert sample interrupt 316 that sets the SI 134in the interrupt register 118 of FIG. 1. Latch 318 can be set andcleared by firmware 120 as sample interrupt enable 320 to preventunwanted interruptions. An AND-gate 322 combines output of the latches318 and 306 to perform enabling and disabling of the sample interrupt316.

The use of the AND-gate 302 prevents new samples in the sample buffer218 of FIG. 2 from being overwritten by sample pulse 220 until thefirmware 120 gives explicit allowance to do so. Besides guaranteeingthat each sample that was taken can be logged out, this also ensuresthat new instrumentation data samples cannot be saved to the samplebuffer 218 while the firmware 120 is in the process of reading thesample buffers 218 across all of the trace array and logic blocks 102.Otherwise, this could lead to inconsistent samples that hold partialdata from two different sample pulses from disparate instances in time.

The actual interrupt indication to the firmware 120 may be active wheneither an instrumentation data sample was stored to the sample buffer218 or when the trace array 204 contents should be saved, as generatedby LTIG 132. To facilitate differentiating the two events, the SI 134and LTI 136 are available in the interrupt register 118 that is quicklyread by the firmware 120 via path 126. In an exemplary embodiment, theSI 134 indicates that an instrumentation data input 112 sample wasstored to the sample buffer 218; the other indicator, the LTI 136,indicates that the trace array 204 contents should be saved.

One example of a sequence performed by the firmware 120 upon receivingan interrupt from the trace controller 104 is depicted in FIG. 4 as partof a process 400 for sampling computer system performance data withoutimpacting RAS of the computer system. At block 402, the firmware 120enables assertion of the sample interrupt 316 via the sample interruptenable 320. At block 404, the sample pulse generator 128 outputs samplepulse 220 to the sample buffer 218. The sample pulse 220 triggers thesample buffer 218 to store a sample of the instrumentation data input112. The feedback signal 216 holds the sample in the sample buffer 218,providing a window of time for the firmware 120 to initiate a read ofthe sample buffer 218. At block 406, the sample interrupt generator 130asserts sample interrupt 316 in response to sampling the instrumentationdata via the sample pulse 220. The sample interrupt 316 indicates thatthe instrumentation data is available to read from the sample buffer 218for measurement of computer system performance through configuration ofthe read mode mux 226 and the byte mux 222. At block 408, the firmware120 receives an indicator of the sample interrupt 316. The firmware 120can further discern the interrupt source by reading the interruptregister 118, e.g., SI 134 or LTI 136, where the LTI 136 is asserted inresponse to filling the trace array 204.

At block 410, the firmware 120 selects a data source to read as thesample buffer 218 in response to receiving the indicator of the sampleinterrupt 316, where the sample buffer 218 stores instrumentation datawhile capturing trace data in the trace array 204. Data from all of thetrace array and logic blocks 120 can be read. At block 412, the firmware102 commands the clear sample interrupt 312 in response to storing thedata to memory 144. The clear sample interrupt 312 de-asserts the sampleinterrupt 316 and allows a new sample pulse 220 to be output forsampling instrumentation data.

Technical effects and benefits include sampling instrumentation datawhile storing trace data. The instrumentation data can be used formeasuring performance of a computer system internally in a processor ofthe computer system with minimal hardware and firmware modifications.This approach eliminates the need to disable hardware tracing whilesampling instrumentation data, thereby addressing RAS concernsassociated with sampling instrumentation in the field, since capturingof data to detect and debug problems can be performed concurrently withsampling performance data.

As described above, the embodiments of the invention may be embodied inthe form of computer-implemented processes and apparatuses forpracticing those processes. Embodiments of the invention may also beembodied in the form of computer program code containing instructionsembodied in tangible media, such as read-only memory (ROM), electricallyerasable programmable read-only memory (EEPROM), flash memory, or anyother computer-readable storage medium, wherein, when the computerprogram code is loaded and executed in a computer, the computer becomesan apparatus for practicing the invention. The computer program code maybe firmware, e.g., firmware 120 of FIG. 1, embedded within an integratedcircuit (IC) chip, such as a processor. The present invention can alsobe embodied in the form of computer program code, for example, whetherstored in a storage medium, loaded into and/or executed by a computer,or transmitted over some transmission medium, such as over electricalwiring or cabling, through fiber optics, or via electromagneticradiation, wherein, when the computer program code is loaded into andexecuted by a computer, the computer becomes an apparatus for practicingthe invention. When implemented in a microprocessor, the computerprogram code segments configure the microprocessor to create specificlogic circuits.

While the invention has been described with reference to exemplaryembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof without departing from the scope of the invention. Inaddition, many modifications may be made to adapt a particular situationor material to the teachings of the invention without departing from theessential scope thereof. Therefore, it is intended that the inventionnot be limited to the particular embodiment disclosed as the best modecontemplated for carrying out this invention, but that the inventionwill include all embodiments falling within the scope of the appendedclaims. Moreover, the use of the terms first, second, etc. do not denoteany order or importance, but rather the terms first, second, etc. areused to distinguish one element from another.

1. A system for sampling computer system performance data, the systemcomprising: a sample buffer to store instrumentation data whilecapturing trace data in a trace array, wherein the instrumentation dataenables measurement of computer system performance; a sample interruptgenerator to assert a sample interrupt indicating that theinstrumentation data is available to read, wherein the sample interruptis asserted in response to storing the instrumentation data in thesample buffer; and wherein the sample interrupt generator dc-asserts thesample interrupt in response to the command to clear the sampleinterrupt.
 2. The system of claim 1 further comprising: a sample pulsegenerator to output a sample pulse triggering the sample buffer to storethe instrumentation data.
 3. The system of claim 2 wherein a command toclear the sample interrupt notifies the sample pulse generator of theread of the sample buffer.
 4. The system of claim 1 further comprising:a read multiplexer, wherein the read multiplexer is configurable tooutput data on a data return path from either the trace array or thesample buffer.
 5. The system of claim 1 further comprising: a log traceinterrupt generator to assert a log trace interrupt upon filling thetrace array; and an interrupt register holding interrupt sourceindicators for the log trace interrupt and the sample interrupt.
 6. Thesystem of claim 1 wherein the sample interrupt generator includes asample interrupt enable latch to enable assertion of the sampleinterrupt.
 7. The system of claim 1 wherein the sample buffer and thesample interrupt generator are integrated in a processor of the computersystem.
 8. A method for sampling computer system performance data, themethod comprising: sampling instrumentation data in a sample bufferwhile capturing trace data in a trace array, wherein the instrumentationdata enables measurement of computer system performance; asserting asample interrupt in response to sampling the instrumentation data,wherein the sample interrupt indicates that the instrumentation data isavailable to read; and de-asserting the sample interrupt in response tothe command to clear the sample interrupt.
 9. The method of claim 8further comprising: outputting a sample pulse to the sample buffer,wherein the sample pulse triggers the sample buffer to store theinstrumentation data.
 10. The method of claim 9 further comprising:receiving a command to clear the sample interrupt, wherein the commandto clear the sample interrupt provides notice of a read of the samplebuffer.
 11. The method of claim 8 further comprising: configuring a readmultiplexer to output data on a data return path from either the tracearray or the sample buffer.
 12. The method of claim 8 furthercomprising: asserting a log trace interrupt upon filling the tracearray; and storing interrupt source indicators for the log traceinterrupt and the sample interrupt.
 13. The method of claim 8 furthercomprising: enabling assertion of the sample interrupt.
 14. The methodof claim 8 wherein sampling the instrumentation data and asserting thesample interrupt are performed in a processor.
 15. A computer programproduct for sampling computer system performance data, the computerprogram product comprising: a computer-readable storage medium storinginstructions for executing sampling computer system performance data,the sampling computer system performance data comprising a method of:receiving an indicator of a sample interrupt, wherein the sampleinterrupt indicates that instrumentation data is available to read froma sample buffer for measurement of computer system performance;selecting a data source to read as the sample buffer in response toreceiving the indicator of the sample interrupt, wherein the samplebuffer stores instrumentation data while capturing trace data in a tracearray; enabling assertion of the sample interrupt; and clearing thesample interrupt in response to storing the instrumentation data to thememory; storing data from the selected data source to memory.
 16. Thecomputer program product of claim 15 further comprising: resetting asample interrupt generator in response to clearing the sample interrupt,wherein the sample interrupt generator controls outputting of the sampleinterrupt.
 17. The computer program product of claim 15 furthercomprising: determining an interrupt source as one of the sampleinterrupt and a log trace interrupt, wherein the log trace interrupt isasserted in response to filling the trace array.
 18. The computerprogram product of claim 15 wherein the sample buffer is one of aplurality of sample buffers and the trace array is one of a plurality oftrace arrays, and further wherein selecting the data source to read isperformed for the plurality of sample buffers and the plurality of tracearrays.
 19. The computer program product of claim 15 wherein theinstructions are firmware in a processor of the computer system.